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Background / Context: 


Beginning with the No Child Left Behind Act of 2001 (NCLB) and reinforced by the 
Education Sciences Reform Act of 2002 (ESRA), the federal government established explicit 
expectations for the role of research in informing decisions about education programs, policies, 
and practice, and it also specified new expectations for what constituted research knowledge that 
was worth using. NCLB legislation went so far as to include in its definition of scientifically- 
based research “a preference for random-assignment experiments” in impact evaluations of 
programs or policies. Following this, ESRA established the Institute of Education Sciences (IES) 
which began pushing the research community (i.e., through grants and contracts) to do more 
randomized experiments (Viadero, 2004). In 2002, IES established the What Works 
Clearinghouse (WWC), which reviews, critiques, and synthesizes evidence of impacts of 
education interventions. Although the WWC got off to a slow start (Viadero, 2006), it now 
includes hundreds of Intervention Reports and Practice Guides based on reviews of more than 
10,000 studies. Now, IES is 15 years into its effort to transform education research, and there are 
clear indications that the education research produced today is quite different from that of twenty 
years ago (NBES, 2008; IES, 2017). Unfortunately, we don’t know much about whether these 
changes have actually led schools and individual educators to make better and more frequent use 
of research to inform their decisions. 


Purpose / Objective: 


This paper documents early results from a nation-wide study of when and how educators 
use research in decisions about policy and practice. To motivate new approaches to increasing 
research use and evidence-based decision-making in schools, our team is conducting a series of 
research studies, the first of which involves developing and validating a survey to measure depth 
of research use by schools and teachers through collection of multilevel data from district staff, 
school administrators, and classroom teachers. In this paper, we present results from the first 
pilot of this survey with over 500 educators from 32 schools. The results presented in this paper 
focus on the influence of scientifically-based research and IES-funded supports for research use 
(e.g., WWC, RELs) on school-wide and classroom-level decisions about practice. 


Setting: 


In spring 2017, a large-scale pilot of the survey was conducted with administrators and 
instructional staff from 32 schools in two states (i.e., one mid-Atlantic and one New England). 


Population / Participants: 


The sampling frame included 1,513 people who were either instructional staff (e.g., 
teachers, coaches) or responsible for supervising instruction (e.g., principals). A total of 580 
people responded to the survey (for an overall response rate of 38%) including 25 school 
administrators, 12 district staff, 51 instructional coaches/specialists, 79 special education 
teachers, 344 additional classroom teachers, and 69 other instructional staff (e.g., ESL teachers). 


Research Design: 


This is a large-scale survey research study with three phases. The first two phases involve 
piloting the survey in separate samples of 30-schools. A nationally-representative field test will 
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be conducted in 2018-19 involving 300 schools. The data for this paper will be augmented with 
data from the second pilot prior to presenting at SREE in spring 2018. 


The survey included two paths, with the path for each respondent determined by whether 
they stated substantial familiarity with a specific organizational decision (Path-A; see Figure 1 
on Appendix page A2). Those who did not have such familiarity were routed to an alternative 
path in which the questions focused on the evidence and information used to select and 
implement a specific “strategy” to support student learning (Path-B; see Figure 2 on Appendix 
page A3). Metaphorically, the instrument was a ‘choose-your-own adventure’ survey in which 
responses to prior questions determined the wording and sequence of questions that appeared 
next. 


Data Collection and Analysis: 


Data were collected through online administration of the survey using the Qualtrics 
platform. Analyses presented here involve simple descriptive statistics (i.e., frequency counts). 


Findings / Results: 


Based on the data from our first pilot, the majority of school administrators (78%) and 
about half of the instructional coaches/specialists (49%) followed Path A, answering questions 
about a school-wide organizational decision that they described via an open-ended question (see 
Figure 1). Within the other respondent groups, between 20% and 29% also followed Path A. The 
remaining respondents were routed to Path B, and answered questions about “a key strategy they 
have used to support students’ progress” (see Figure 2). In total, there were 155 responses to the 
survey items about evidence influencing organizational decisions, and there were 308 responses 
to the survey items about evidence influencing individuals’ decisions. 


The figures on Appendix page A4 present bar charts showing frequency of responses to 
individual survey items for Path A (i.e., focused on school-wide organizational decisions). 
Results suggest that the majority of organizational decisions relied moderately or heavily on “a 
claim about the impact of a program or policy,” and that the majority of respondents believed 
that the decision was informed by scientifically-based research. However, the WWC, RELs, and 
Federal/State Departments of Education almost never informed these decisions. 


Similar figures on Appendix page A5 present bar charts showing frequency of responses 
to individual survey items for Path B (i.e., focused on individual educators’ selection of 
strategies). Results suggest that the majority of strategies relied moderately or heavily on “a 
claim about the impact of a program or policy,” and that the majority of respondents believed 
that the strategy is supported by scientifically-based research. In comparison to the results for 
organizational decisions, the WWC and RELs exhibited even weaker influence on selection of 
strategies. Additional data included in our presentation will show just which sources practitioners 
rely on most. 


Conclusions: 


There are many reasons why research highlighted by the WWC, RELs, and DOEs might 
have infrequent and limited influence on decisions in schools and classrooms. This paper serves 
to highlight the likely scope and severity of the apparent divide between educators’ decisions and 
the effectiveness research intended inform those decisions. We hope that our presentation at 
SREE would promote conversations about the research<+practice divide, and how we might 
address it. 
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Figure 1. Path-A routing question about an organizational decision. 


The following question focuses on organizational decisions, which are decisions about policy and 
practice made at the school or district level that affect a significant number of teachers and/or 
students. For example, some organizational decisions a school district might make include (a) 
adopting or discontinuing a program or intervention, (b) adopting a new curriculum, and (c) providing 
specific PD training to teachers. 


Please think of a major organizational decision related to student outcomes that was made by your 
school or district during this school year or the previous school year. Please describe this decision in a 
few sentences addressing the following points: 


Q5_1. What was the problem that the decision was intended to address? 


Improve retention of new teachers. 


What was the decision (i.e., what was changed or introduced/what actions were taken)? 


The district implemented an induction program for new 
teachers. 
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Figure 2. Path-B alternate question about an individual decision. 


Because your responses suggest you are not deeply familiar with an organizational decision, we would 
like to ask you instead about key strategies you have decided to use in your own practice. 


Please identify a key strategy or approach that you have used recently to support students’ 
progress during this school year or the previous school year. Please describe this strategy or 
approach in a few sentences addressing the following points: 


What was the strategy or approach? 


What was the purpose or goal of this strategy/approach? 
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Figures showing frequency of responses to individual survey items for Path A 
(i.e., focused on school-wide organizational decisions). 


Five Randomly-Selected Responses to “What was the 
decision (i.e., what was changed or introduced/what 
actions were taken)?” 


e _ It was approved to move forward with all day 
kindergarten throughout the district. 


e Weimplemented the use of new resources within 
our reading support classrooms 


e To make the school year start sooner 


e Acompletely overhauled district report card 
specifically tailored for individual grade levels. 


@ adopting a new curriculum 


To what degree did the information that informed this decision rely on... 
a claim about the impact of a program or policy (i.e., “Did it work?”) 


Not at all 


Somewhat 


Moderately 


Heavily 


T T T T T T T T T 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Percent of Frequency 


To what degree did the information that informed this decision rely on... 
a scientifically-based research study 


Not at all 


Somewhat 


Moderately 


Heavily 


T 


T T T T T T T T T 
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Percent of Frequency 
Which of the following sources provided information, and to what extent did 


the information influence the decision? ... 
A Regional Education Lab or Comprehensive Center 


Provided No Information 


| Don't Know/Remember 


Provided information that had little or no influence 


Provided information that had some influence 


Provided information that heavily influenced the decision 


T T T T 
0% 20% 40% 60% 80% 


Percent of Frequency 


100% 


Which of the following sources provided information, and to what extent did 
the information influence the decision? ... 
The What Works Clearinghouse 


Provided No Information 


| Don't Know/Remember 


Provided information that had little or no influence 


Provided information that had some influence 


Provided information that heavily influenced the decision 


T T T T Tt 
0% 20% 40% 60% 80% 100% 


Percent of Frequency 


Which of the following sources provided information, and to what extent did 
the information influence the decision? ... 
Guidance from federal or state departments of education 


Provided No Information 


| Don't Know/Remember 


Provided information that had little or no influence 


Provided information that had some influence 


Provided information that heavily influenced the decision 


T T T T 
0% 20% 40% 60% 80% 100% 


Percent of Frequency 


SREE Conference Abstract Template 


Figures showing frequency of responses to individual survey items for Path B 
(i.e., focused on individual educators’ selection of strategies). 


Five randomly selected responses to the survey question, “What was 
the strategy or approach?” 


e¢ | use a lot of modeling with concrete objects 


e When working with students who are English Language 
Learners (ELLs), | use the approach of providing the students 
the opportunity, when possible, to make connections with 
their own culture/language and share those connections with 
the rest of the studentsin the small group (most of whom are 
not ELLs). 


e | use differentiated instruction to meet the needs of my 
students. 


e — Having special education students more included into general 
education specials and classes. 


@ = flexible group instruction for math 


To what degree did the research that supports the use of this strategy rely on... 
a scientifically-based research study 


Moderately 
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Where did you get information to support your use of this 
strategy/approach? ... 
A Regional Education Lab or Comprehensive Center 


T T T T T T 
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To what degree did the research that supports the use of this strategy rely on... 
a claim about the impact of a program or policy (i.e., “Did it work?”) 


Not at all 


Somewhat 


Moderately 
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Percent of Frequency 
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Where did you get information to support your use of this 
strategy/approach? ... 
The What Works Clearinghouse 


Yes 
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Where did you get information to support your use of this 
strategy/approach? ... 
Guidance from federal or state departments of education 
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